Controlling an Agent by Focusing its Attention on Interactivelly Selected Patterns

Designing robotic controllers can quickly become a challenging problem. Indeed, such controllers have to face a huge number of possible inputs that can be noisy (e.g. think of visual inputs as furnished by cameras), to select actions among a continuous set, and to automatically adapt themselves to evolving or stochastic environmental conditions. In realworld robotic applications, it is difficult to model the environment formally and to specify how to solve a given task directly in a programming language. On the other hand, living beings face such problems everyday, and are able to learn how to solve them efficiently. Some research works have therefore tried to mimic natural strategies for solving robotic tasks. Neuropsychological evidence shows that the natural learning process is interactive: after each interaction with the environment, the agent gets a reward signal, called the reinforcement, which measures its performances. The goal of the agent is to maximize its reinforcements over time. As the agent observes the consequences of its reactions on the obtained reinforcements, it gains more and more expertise on its task, which ultimately enables it to act optimally. It is important to note that in such a setup, the agent is never directly told the optimal action to choose: there is no explicit external teacher. As an example, consider the task of a robotic agent escaping from a discrete maze: the reward could be zero until it manages to find the exit of the maze, in which case it obtains a positive reinforcement. Note that in the latter task, the interest of doing some action can appear only a long time after the interaction. This is called the delayed reward problem. Those ideas gave rise to the algorithmic theory of Reinforcement Learning (RL) [1], the goal of which is to solve closed-loop adaptive control problems without a model by analyzing the reinforcements earned during a sequence of interactions. The major advantages of the RL protocol are that it is fully automatic, and that it imposes weak requirements on the environment. RL can be applied in any environment that obeys a discrete-time Markovian probabilistic dynamics, and upon which it is possible to define a reinforcement function r(s,a) that gives for each state s of the environment and each action a, a numerical measure of the worthiness of doing this action in this state.